Some individuals may not be able to interact with a computer user interface the way it is commonly used. For instance, small icons and type pose a challenge for the visually impaired. Audible alerts and feedback are useless to the hearing impaired. The computing industry is sensitive to these needs. Some operating systems come with additional accessibility features that enable those with disabilities to modify the user interface in ways that are more accommodating to their needs. For instance, some operating systems allow users to enable visual feedback where audible feedback would otherwise be used. In addition, extra large screen fonts and high contrast schemes may used for users with low vision. For those with extreme visual impairments, such as the blind, some operating systems provide “screen readers” that narrate the elements of the user interface to the user or provide infrastructure allowing another company to provide such a screen reader.
A typical screen reader utility executes concurrently with whatever application the user may be working with. As the user navigates from element to element, such as by tabbing from one button to another, the screen reader sends information about the current element to a text-to-speech engine and/or a refreshable Braille display to convey that information to the user. Text-to-speech engines translate this information into synthesized speech to announce it to the user. Refreshable Braille displays translate that information into a well-defined pattern of dots (i.e., Braille characters) and raise pins on a physical hardware device corresponding to each dot in the Braille characters. In the case of a button, the screen reader often conveys the name of the button and the current state of that button (e.g., it is currently disabled and therefore cannot be pressed). Similarly, if a user is in a word processing application, the screen reader can be configured to identify the foreground window (i.e., name of the application) and the current line, sentence, word, or character closest to the insertion point. The screen reader can also describe attributes of that text, such as the font name, weight, color, emphasis, and justification. Often times, the screen reader also informs the user what actions the user may currently take. For instance, if the user has navigated to a button, the screen reader may notify the user that they may press the button by tapping the space bar.
Screen readers are indispensable for computer users with certain visual impairments. In general, many users would simply not be able to take advantage of a computer without an assistive technology product that compensates for their loss of mobility, sensory perception, or other facilities that can be enhanced through technology. However, current software design methodologies make assistive technology products, such as screen readers, difficult to design. As mentioned, the assistive technology product typically receives a notification of a change to a currently-running application or the operating system environment itself. Often this notification takes the form of an event indicating that focus has changed from one element (e.g., a button or list box) to another element (e.g., an edit field, icon, or the like) or that a new element has been created or destroyed (e.g., a window has been opened or closed). A selection manager associated with the application raises the event and notifies the operating system of the change. In response, the assistive technology product may query the selection manager to determine what element is associated with the event (e.g., which element has the focus) so it may obtain additional information to convey to the user.
Currently, assistive technology products essentially are only able to request from the element a limited set of information such as its type (e.g., button, list box, or the like), its location on the screen, or its caption. The assistive technology product itself must then deduce from the returned element type what functionality is available to the user. In other words, the assistive technology product must understand what a “button” is and that the button may be pressed (invoked). Therefore, the designers of a good assistive technology product must predefine all of the types of elements that might be included in an application and identify their functionality. This is an impossible task because there are new types of screen elements or controls produced on a routine basis by software companies throughout the software industry. In addition, this is an inefficient use of resources because not all elements are unique. Many elements share similar functionality, such as the ability to be invoked or the ability to manage a collection of items where one or more items may be selected.
A more general class of applications, automation utilities, has nearly the same set of requirements as these assistive technology products. In general, automation utilities need the ability to dynamically discover screen elements (e.g., controls) whether by traversing the object hierarchy of elements or by receiving an event notification, such as when the focus changes from one control to another. These utilities also need a general mechanism for querying these elements for human-readable information that can be conveyed to the user or stored for later reference. Finally, automation utilities need the ability to discover what functionality or behavior is offered by a particular screen element, even when the element is completely unknown to the automation utility. Unfortunately, a superior mechanism for discovering elements of a user interface and querying and manipulating their associated functionality in such a way that it can be applied to the full spectrum of possible elements has eluded those skilled in the art.